Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

LGTM, @WenyXu PTAL

I came up with two scenarios of how JSON would be used by end users:

• Data source uses JSON to encode their data, but the schema to JSON is relatively fixed. Like OTLP protocol or some HTTP header/response. In this case the JSON itself is usually defined with a clear schema.
• Use us to store an arbitrary JSON object. Neither the user nor user of our user knows the schema.

For the first case, we can suggest users decompose the JSON object into table columns. Either by themselves (predefined table schema) or by GreptimeDB (gRPC auto alter or through pipeline). So from the DB side, it's just a wide table with several atomic types. And is not distinguishable from other tables. Advantages like high compression ratio or SQL interface are available out-of-box.

The second case on the contrary, requires a fully dynamic ability to JSON data. We'd prefer to handle the "JSON part" on read (like decoding and/or field accessing) to minimize the affection to write performance. This choice is based on the hypothesis that data without known schema is unlikely to be queried frequently.

Reflecting to the implementation phase, the need & no-need to are:

• Need a mechanism to decompose JSON and trigger ALTER TABLE if necessary.
  • This operation needs to happen on frontend or before frontend. The Pipeline is a good choice.
• Need a bunch of util functions to handle (undecomposed, the second case) JSON data
• Don't need to be aware of the concrete JSON schema in the storage engine layer. JSON data is either decomposed for the first case or stored as an opaque field for the second case
• Don't need to introduce STRUCT type as it's not a hard requirement. I.e., it can bring a better experience in expressing the schema of decomposed JSON data, but we can still use this without STRUCT type.

Thanks for all suggestions! I've changed the rfc to a relatively simple jsonb plan which is a first consensus for general json data type. It's not a detailed plan. I think I can first make some efforts in the direction and then get back and refine this rfc.

For those "util functions", we can use https://github.com/datafusion-contrib/datafusion-functions-json which provides an out-of-box implementation. As well as the binary representation, JSONA.

For those "util functions", we can use https://github.com/datafusion-contrib/datafusion-functions-json which provides an out-of-box implementation. As well as the binary representation, JSONA.

set of functions for querying JSON strings in DataFusion

This crate just provides functions for querying JSON strings.

I happens to trigger some auto-review-request because of misoperation QAQ, sry for that.